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I . INTRODUCTION 



A. BACKGROUND 

"The Puny Payoff from Office Computers." That was the 
cover story for Fortune Magazine 26 May 1986. Why is this? The 
computer age was to bring increases in white-collar 
productivity as earlier generations of technological 
innovation did for other sectors, e.g. farming and 
manufacturing. So far, business investment of hundreds of 
billions of dollars in computer systems has failed to 
significantly improve productivity. There has been little or 
no change in white-collar productivity since the sixties, 
where 85-90 percent of information technology is invested 
{Fortune 26 May 1986) . Combined with the fact that 
"information technology is now the largest line item in 
Corporate America's capital spending budget ... Companies need 
to develop better ways of measuring office productivity and 
the contribution that managers and other white-collar workers 
make to the bottom line." {Business Week, 26 November 1990) 

B. OBJECTIVES 

Evaluation of information systems is necessary to 
effectively manage them as organizational resources. To do 
this an organization must be able to measure the costs and 
benefits related to its systems. This allows decisions on 
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design and implementation to "be made in the same way as every 
other firm investment decision, i.e. based on its contribution 
to the success of the firm." (Marsden and Pingrey, 1988) 

Various evaluation methods have been used to assess 
information system effectiveness. The one of special interest 
for this thesis is user satisfaction. A number of articles in 
IS literature cite user satisfaction as a surrogate for 
effectiveness (Ives, Olson and Baroudi, 1983; Pearson, 1977; 
Seward, 1975; Hamilton and Chervany, 1981) . The desire for a 
surrogate stems from the fact that unlike transaction 
processing systems, data on IS use provides little information 
on the effectiveness of the white collar world. The issue here 
is whether opinions about a system such as user satisfaction, 
are the same thing as effectiveness of a system. MIS is a 
multidisciplinary field spanning technical sciences, 
management science, psychology and other social sciences. 
Problems can arise when techniques, theories or results from 
a study in one field are used in another. Management science 
has had difficulty in establishing a link between satisfaction 
and productivity. This thesis will review the MIS literature 
that attempts to show a relationship between user satisfaction 
and system effectiveness. It will examine the development of 
this relationship in the MIS literature. Specifically, it will 
seek answers to the following questions: 

1. Is user satisfaction a useful measurement of IS 
effectiveness? 
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2. What are the various meanings of user satisfaction 
in IS literature? 



3. 



Does the IS literature 
user satisfaction is 
effectiveness for IS, 
previous literature? 



now assume that 
equivalent to 
if so, does it 



measuring 

measuring 

reference 



4. Are other methods used to measure effectiveness? 
What are they? Does the author give a reason for 
the choice? 



C . SCOPE 

This thesis will investigate the measurement of 
information system effectiveness over the past 20 years. It 
examines what methods have been employed in the MIS 
literature. In particular, attention is focused on the use of 
user satisfaction as a surrogate for effectiveness by the MIS 
community. Specific areas of interest include: 



1. What method of measuring system effectiveness was 
employed? 

2. How user satisfaction was initially 
determined/ defined . 

3. The evolution of the user satisfaction- 
effectiveness relationship and its current state. 

4. How and why user satisfaction is employed as a 
surrogate for effectiveness. 

5. Whether the employment of user satisfaction was 
justified, i.e., correct. 



The data gathered will provide information as to the 
various methods of measuring IS effectiveness as well as 
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information regarding the recent prevalence of user 
satisfaction as the preferred method. 

D . METHODOLOGY 

A literature review will be undertaken to document the 
development of the user satisfaction-effectiveness 
relationship and the use of user satisfaction as a surrogate 
for IS effectiveness. It attempts to trace the origin of the 
relationship through to its current state. Attention will be 
focused on these questions: 

1. How was effectiveness measured? 

2. Were other methods cbnsidered and if so were there 
references cited? 

3. If user satisfaction was chosen: 

a. Was the purpose to show user satisfaction as 
equivalent to effectiveness? or, 

b. Did the study assume equivalence and if so did 
it provide a reference? 

Defining and classifying what user satisfaction is and is 
not and what it can and cannot be used for was accomplished by 
drawing analogies from the sociological research by Denzin 
(1989) and Phillips (1971) focusing in particular on the 
descriptions and limitations of the survey technique. This 
parallel was felt justified based on the common psychological 
nature of the measure and the technique used for its 
employment, the survey, between sociological research and the 
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user satisfaction measure of IS research. Basically, it was 
determined that user satisfaction as an opinion, i.e., 
subjective measure should not be employed as an equivalent to 
effectiveness, an action. 

E. BENEFITS 

User satisfaction has aggressively been developed as the 
primary method of measuring IS effectiveness. MIS needs to 
know if this is a proper measurement of its work. What we know 
about information systems we know through the methods by which 
we observe them and the concepts by which we mentally model 
them. If the concepts are erroneous we are vulnerable to 
investing in systems that, in reality, are ineffective. 
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II. BACKGROUND 



A. WHY MEASURE IS EFFECTIVENESS? 

"Information technology is now the largest line item in 
Corporate America's capital spending budget, accounting for 
nearly a third of current-dollar equipment outlays." {Business 
Week, 26 November 1990) The need to understand and measure 
information system effectiveness results from the fact that 
"U.S. business has spent hundreds of billions of dollars on 
them [computers], but white-collar productivity is no higher 
than it was in the late sixties." {Fortune, 26 May 1986) 

Organizations have invested in information technology to 
increase productivity, specifically white-collar productivity. 
"Economists calculate that about 85% of investment in 
information technology is made by the service [white-collar] 
sector. Yet non-manufacturing productivity has risen 
cumulatively just 1.9% over the past 10 years. ... 
'Unfortunately, ' observes Morgan Stanley economist Stephen 
Roach, 'the service sector has devoted far more effort to 
deploying computers than in determining the precise ways they 
can be used to enhance white-collar productivity' 
Companies need to develop better ways of measuring office 
productivity and the contributions that managers and other 
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white-collar workers make to the bottom line." (Business Week, 
26 November 1990) 

Clearly then, increasing the effectiveness of information 
systems is a primary objective of business organizations. To 
do so, they need to know what factors affect and contribute to 
system effectiveness. It is this understanding of 
effectiveness which MIS research must provide so that 
increases in productivity and thus profits are realized, 
thereby justifying the current magnitude of investment in 
information systems by U.S. businesses. 

1 . The Goal of MIS Research 

The goal of research is to increase the understanding 
of some relationship or situation. The goal of MIS research is 
to better describe and understand the factors and 
contributions of information systems to an organization. "MIS 
research is the systematic investigation of the development, 
operation, use and/or impact of an information (sub) system in 
an organizational environment." (Ives, Hamilton and Davis, 
1980) . The goal of information systems is to increase the 
effectiveness of the organization, where effectiveness is the 
ability of performance to meet the goals and objectives set by 
the organization. The purpose of evaluation is to assess or 
appraise the value of an activity and measure its 
effectiveness (Arnovick and Gee, 1978). 
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How MIS measures effectiveness is critical to the 



evaluation process and ultimately the overall understanding of 
information systems. 

2 . Measuring Effectiveness 

Hamilton and Chervany (1981) present two different 
views of system effectiveness. Each has its own meaning of 
system effectiveness and how it should be measured. In the 
goal-centered view "Effectiveness is determined by comparing 
performance to objectives." An example of this approach would 
be cost-benefit analysis. The second is the system- resource 
view where effectiveness is seen as "attainment of a normative 
state, e.g., standards for 'good' practices" rather than 
establishing specific task objectives. 

The complementary nature of these views is similar to 
that of summative and formative evaluation approaches. The 
focus of summative evaluation is the accomplishment of 
objectives. These are tangible factors capable of objective 
measurement. Formative evaluation is assessment of quality and 
consist of intangible factors measured subjectively. "Both 
formative and summative evaluation approaches are typically 
used in providing evaluative information on system 
effectiveness." (Hamilton and Chervany, 1981). 

It is important to point out the significance of the 
dual nature of the evaluation process and the contributions 
and limitations of each approach. 
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a. Goal-Centered View 



Basically the goal-centered view of system 
effectiveness focuses on the efficiency of the system. It 
assesses system performance with respect to established system 
and/or organizational objectives. Quantifiable (tangible) 
benefits provide objective measures of performance. Like the 
summative evaluation approach, the goal-centered view provides 
"information on the systems outcomes, or ends, to support 
decisions to continue, adopt or terminate the system." 
(Hamilton and Chervany, 1981) . That is, the objectives set by 
the organization are the critical factors against which the 
system success or failure is judged. 
b. System-Resource View 

The system-resource view uses subjective measures 
of intangible benefits to assess the effectiveness of the 
system. Like the formative evaluation approach, the system- 
resource view provides supplementary /explanatory information 
about the system (Hamilton and Chervany, 1981), 

B. THE REASON FOR THIS STUDY 

Hamilton and Chervany' s (1981) dual view of system 
effectiveness identifies these two system evaluation 
approaches as separate yet complimentary. This complimentary 
relationship is analogous to the one between summative and 
formative evaluation approaches where information provided by 
the formative evaluation helps "improve the means and process 
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to accomplish objectives and interpretation of summative 
evaluation results." They further explain (1981) that 
objective measures are provided by the "formal evaluation 
approaches" while "perceptions of system effectiveness" are 
gained by the informal ones which are "necessary and helpful 
for calibrating the credibility of information on MIS 
evaluation . " 

Assessing the accomplishment of official goals and 
objectives and thus the overall effectiveness, or success, of 
the system is the function of the formal evaluation 
approaches, i.e., goal-centered view, summative evaluation, 
and objective measures. The informal approaches/ the system- 
resource view, formative evaluation, and subjective measures 
provide supplementary information, recognizing that "systems 
have other functions and have other consequences besides 
accomplishment of official objectives" (Hamilton and Chervany, 
1981) . 

The message here is that the system- resource view, 
formative evaluation and/or subjective measures are informal 
approaches and are at best supplementary measures to "explain" 
or "help ... and aid." The point being that the goal-centered 
view, summative evaluation and objective measures are formal 
evaluation approaches and are the primary means of determining 
the success or failure of the system, to be supplemented by 
their "informal" counterparts. 
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Relevant to the current study is the subjective measure, 
user satisfaction, which is a perception of system 
effectiveness. Although Hamilton and Chervany suggest that 
"The user attitude survey [user satisfaction] is a formal 
approach to quantify subjective perceptions of system 
effectiveness." this thesis contends that user satisfaction is 
none the less just a supplementary measure of system 
effectiveness . 

The hazard in employing user satisfaction as a prime means 
of evaluating the effectiveness of information systems, to the 
exclusion of assessing costs and benefits, is that it is an 
opinion-based technique. An example of the hazard is 
illustrated by international surveys of students' opinions of 
their math competency. In such surveys, U.S. high school 
students rate themselves among the worlds best. In actual 
testing of math skills, U.S. high school students are among 
the world's worst. The opinion is not the same as the skill. 
Similarly, users of information systems may have numerous 
motives to claim satisfaction with these systems. Yet, these 
opinion reports are not, conceptually, the same thing as an 
assessment of a system's productivity in terms of inputs and 
outputs or its quality in terms of better management 
decisions. Indeed, in two studies where both effectiveness and 
satisfaction were measured, the findings were contradictory, 
Gallupe and DeSanctis (1988), and Aldag and Power (1986). It 
is the distinction between primary and supplementary measures 
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that will serve as criteria for classifying articles reviewed 
in this thesis. The purpose will be to trace the genesis and 
evolution by which user satisfaction crowded out cost-benefit 
analysis . 
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III. METHODOLOGY 



A. SELECTION OF SYSTEM EFFECTIVENESS STUDIES 
1. Selection of Articles for Study 

The concept of this study has as its impetus, 
questions raised in a previous thesis on information system 
effectiveness by Mustofa (1989). Therefore, it seemed 
reasonable to begin article selection with those identified as 
dealing with systems effectiveness in the preceding study. To 
these were added articles published after 1988, the last year 
included in the Mustofa study. These new articles were 
collected by the same means, which will be briefly discussed. 
(The reader may refer to Chapter III Methodology, Mustofa 
(1989) for a detailed account of the search and selection 
process . ) 

A consensus of journal names from three studies 
reporting on articles published in the MIS field was selected. 
The starting point was 1970 and continued through 1988. A 
DIALOG search from the INSPEC database was conducted by the 
Dudley Knox Library at the Naval Postgraduate School. The 
bibliographic search was on the following key: (Productivity 
or Effectiveness) and (Measures or Measurement or Assess or 
Assessment) and Information System. This automated search 
turned up a limited number of useful articles, fewer than 25. 
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Therefore, a manual search was conducted examining the 
periodicals on the consensus list between the years 1970 and 
1988. Indices or every issue of periodicals without an index, 
were examined for articles possibly dealing with information 
system effectiveness. This search turned up 118 articles as 
well as the reason for the small number found during the on- 
line search, the limited number of key word synonyms for 
effectiveness and productivity. Additionally, articles were 
obtained from friends and colleagues aware of the situation or 
involved in related work as well as those published subsequent 
to 1988 up to July 1990. 

2. Methods of Evaluating System Effectiveness 
a . Knowledge 

Knowledge accumulation is an important aspect in 
"the relationship between actions and outcomes" (Goldstein et 
al., 1990) . Various methods of knowledge accumulation have 
been proposed and summarized by Goldstein et al. (1990) : Weick 
(1979) believes that "individuals make sense of events 
retrospectively," that is, they assimilate information about 
the event (s) and then explain them retrospectively to 
understand them. Salancik and Porac (1986) have identified a 
three step process; gather data to understand cause and effect 
relationships, build a knowledge base of these relationships, 
and use them to come to generalized statements. Isenberg 
(1986) identified "a cycle of reasoning and action" using 
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existing knowledge identify any new information required and 
then act and update through feedback. Lord and Foti (198 6) 
describe a knowledge representation scheme classifying 
different environmental elements which are used as prototypes 
and organized hierarchically for decision making. 

The amount of knowledge accumulated is linked to 
job performance. Experts were found to have not only more 
information available to them but it was also more useable 
i.e., they had more and better descripters, and more inter- 
category information (Lurigio and Carroll, 1985) . Goldstein et 
al. (1990) used a knowledge classification scheme of 
declarative, procedural and causal knowledge. "Declarative 
knowledge consists of facts and definitional descriptions of 
categories . . . Procedural knowledge is the heuristics and 
rules that guide a person's actions ... Causal knowledge 
consists of the description of the existence, strength and the 
direction of the relationships among categories and 
strategies . " 

b. Cost-Benefit-Analysis 

"A cost-benefit-analysis (CBA) is essentially a 
comparison between two states. Proposed new system costs and 
benefits are usually compared with those of current systems 
whether they be manual, partly computerized or fully 
computerized." (Lincoln, 1986). There is usually little 
difficulty identifying costs of the systems as these are 
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usually well documented. However, identification of benefits 
and subsequently assigning a value to them has proven to be 
difficult. Basically there are two types of benefits, tangible 
and intangible. Tangible benefits are relatively easy to 
quantify and assign a value. Intangibles, on the other hand, 
have proven to be hard to quantify and when values were 
assigned they have been viewed skeptically. Often they are not 
even considered in the problem. There are different 
philosophies to the valuation of intangible benefits. Matlin 
(1979) argues for "placing dollar amounts on all intangible 
benefits" stating that "Difficulty in finding a value is not 
an appropriate reason for avoiding the subject. It is vitally 
important to derive techniques for valuing benefits of 
information system investments. Even an approximation of the 
values will provide managers an opportunity to make informed 
decisions about future investments. Without these values 
decisions may be made without appropriate information." 
(Matlin, 1979) 

Another approach is made by Hall and Lincoln 
(1976). They consider only quantifiable benefits, estimate 
them conservatively and check "the financial results for 
accuracy and plausibility" reasoning that senior management 
has been made skeptical of CBA by "many examples of quick, 
glib, cost justification exercises." 

The value obtained is then used in a comparison 
between pre and post system introduction, but requires a 
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stable environment or a control group in place to be a valid 
measure. Alternatively, the value may be compared against 
another system, which however, requires "a reconstruction of 
the state of affairs assuming either an alternative system is 
in place or the activity was not undertaken." (Lincoln, 1976) . 
Determining the basis for comparison i.e., previous system, 
alternative technology or manual system and the level of 
service is complicated, difficult and uncertain. Senior 
executives often require initial comparison to a manual 
system. Its advantages include zero-based comparison and 
familiarity due to a large database of similar applications 
over many years. 

CBA has been criticized because of the difficulty 
and subjectivity in identifying and/or evaluating the 
intangible benefits of unstructured, ad hoc decisions and 
obtaining determinable data supporting system success but 
unrecorded by an organization (Ives et al., 1983). 
c. Value Analysis 

Value analysis is similar to user satisfaction upon 
which it is based but is preferred in evaluating DSS (Money et 
al., 1988). Value analysis is employed in evaluating DSS and 
begins with the premise that "information is data of value in 
decision making" (Yovits and Foulk, 1985) . They also suggest 
"that information gives rise to observable effects through the 
decision making process and information feedback exists so 
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that the decision maker (DM) will adjust his assessment of the 
decision situation for later decisions" (Yovits and Foulk, 
1985) . Money et al . (1988) using the method of conjoint 
measurement cohvert subjective judgement of the value of 
intangible benefits into scores for analysis. 

d. Profit Maximization 

A key objective of an organization is profit 
maximization. Information systems design should focus on 
achieving that goal. Marsden and Pingrey (1988) argue that 
decisions regarding IS be made as any other investment 
decision, "its contribution to the success of the firm." They 
develop a goal drive mathematical process model which seeks 
the goal of profit maximization. Using values of revenue 
enhancement or cost reduction of various information products 
and costs of the necessary man-machine inputs of those 
products, they maximize the difference. Citing numerous 
references in the economics and IS fields to support their 
technique. Cron and Sobol (1983) use multiple documented 
financial performance measures; pretax return on assets, 
return on net worth, pretax profits as a percent of sales and 
average five-year growth as indicators of performance, i.e., 
effectiveness . 

e. Use 

Use of a computer system is expected to improve 
performance of its users (Lucas, 1974). Therefore the 



18 



relationship between use of a system and performance 
(effectiveness) is investigated. Effective systems are assumed 
to attain higher use than less effective ones. However, this 
requires that use of the system is optional as for a DSS. With 
mandatory use of the system the relationship between use and 
effectiveness fails. Criticism of use as a measurement for 
effectiveness comes from the fact that it measures only the 
input to the system, where as the desired value 
(per formance/ef feet iveness ) concerns the output of the system. 

f. Productivity 

Productivity as a measure of system effectiveness 
is a before and after comparison of some performance 
objective (s) . There are various levels of sophistication in 
defining these objectives, from measuring simple letter 
production using a word processing system (Kapsales, 1986) to 
sophisticated regression models of multiple variables 
(Englander and Englander, 1985) . 

g. User Satisfaction 

User satisfaction is a measure of the users 
subjective evaluation of factors concerning an information 
system's utility. These factors originate from work on 
organizational behavior by Cyert and March. They argue that 
the users "need for information" is either reinforced or 
frustrated by the information system, the success 
(effectiveness) of which is the users sense of satisfaction. 



19 



Most user satisfaction instruments 



use the semantic 



differential technique developed by Osgood, Suci and 
Tannenbaum, e.g., the Shult z-Slevin questionnaire. However, 
"Most of those that existed prior to an empirically derived 
methodology developed by Pearson (1977) have been heavily 
criticized" (Evans et al., 1988). Bailey and Pearson (1983) 
developed a tool from Pearson's 1977 dissertation, combining 
factors from "a review of 22 studies of the computer/user 
interface." The Bailey and Pearson tool is based on Wanous and 
Lawlers model defining satisfaction as "the sum of the user's 
weighted reactions to a set of factors, 

' E Xi, 

i = l 



where 

Rij = The reaction to factor j by individual i. 

Wij = The importance of factor j to individual i. 
This model suggests that user satisfaction is the sum of ones 
positive and negative reactions to a set of factors." (Bailey 
and Pearson, 1983) . Bailey and Pearson also utilize the 
semantic differential technique of Osgood et al . 

Ives et al. (1983) reduced the Pearson tool to "a 
'short form' of the instrument for use when time is limited 
and/or only an overall assessment of UIS [user information 
satisfaction] (rather that analysis of particular symptoms of 
a problem) is adequate." 
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h. Multiple Criteria 

The general intent of multiple criteria studies is 
to more thoroughly "examine the significance of various 
information system characteristics" (Dickson et al., 1977) for 
"an easy-to-use instrument for improving the effectiveness of 
information systems" (Evans et al., 1988). Proponents of the 
multiple criteria approach conclude that "Information systems 
are a product of multiple factors" (Mansour and Watson, 1980) . 
In general, some combination of system performance, goal 
attainment and user criteria is employed. The intent being to 
more fully describe and define the IS environment, thereby 
more accurately evaluating the IS effectiveness (Mansour and 
Watson, 1980; Chandler, 1982; Evans et al., 1988). Chandler 
suggests information systems be evaluated from two 
perspectives, one being the computer system domain the other 
the user domain, each having its own goals and measures. Evans 
et al. conclude that by applying different methodologies the 
various results "can work separately and in harmony with each 
other" (Evans et al., 1988). 

3. Evolution of the User Satisfaction Tool 

Cyert and March (1963) are credited by Bailey and 
Pearson (1983), and Ives, Olson and Baroudi (1983) with being 
the source of the user information satisfaction (UIS) concept. 
Basically, they argue that "when an organization discovers a 
solution to a problem by searching in a particular way, it 
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will be more likely to search in that way in future problems 
of the same type; when it fails to find a solution by 
searching in a particular way, it will be less likely to 
search in that way in future problems of the same type." 
Drawing inference from this to information systems, it is held 
that the ability of the system to meet the needs of the user 
either reinforces or frustrates the users' sense of 
satisfaction with the system. Therefore, it is suggested that 
"Satisfaction of users with their information systems is a 
potentially measurable, and generally acceptable, surrogate 
for utility in decision making" (Ives, Olson and Baroudi, 
1983) . 

User satisfaction is a subjective measure of system 
success. It relies on the users perceived value of the 
information system's effectiveness to be a surrogate measure 
for objective measures deemed to difficult to obtain. 

a.. Early User Satisfaction Measures 

The early measures of user satisfaction, those 
through 1977, had varied terminology for the concept of user 
satisfaction, for example; "appreciation" (Swanson, 1974), 
"user attitudes" (Lucas, 1974), "user perceptions" (Gallagher, 
1974; Schewe, 1976), "perceived success" (Edstrom, 1977), 
"user evaluation" (Dickson, Senn and Chervany, 1977). 
Generally these measures were uniquely "designed" for the 
particular study that employed it. Often the definition and 
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construct of the variable were not described in the article, 
see Appendix E, Table VII. 

b. The Shultz-Slevin Questionnaire 

The Shultz-Slevin Questionnaire is based on the 
Likert and semantic differential techniques. The Likert 
instrument "consists of a number of statements to which the 
respondent indicates how strongly he agrees or disagrees on a 
five-point or seven-point scale" (Shultz and Slevin, 1975) . 
The semantic differential is a scale that "consists of the 
statement of a concept to which an individual responds by 
checking a set of bipolar adjectives; good-bad." The Shultz- 
Slevin Questionnaire became the prevalent method in the 
studies reviewed frgm 1978 to 1983. The term user satisfaction 
became more common, although each study generally still had 
its own definition of "user satisfaction." However, the 
construct of the measure began to standardize around the 
Shultz-Slevin Questionnaire, see Appendix E, Table VII. 

c. The Pearson Methodology and the Bailey and Pearson 

Instrument 

Examining Table VII with respect to studies 
published after 1983 shows that either the Pearson Methodology 
or the Bailey and Pearson Instrument was almost the exclusive 
measure of user satisfaction when evaluating information 
system effectiveness. The Bailey and Pearson Instrument is 
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described in section (2), Methods of Evaluating System 
Effectiveness . 

Evans et al. (1988) sum up the evolution of user 
satisfaction this way: 

Instruments to measure user satisfaction are scarce. Most 
of those that existed prior to an empirically derived 
methodology developed by Pearson (Pearson, 1977) have been 
heavily criticized (Ives, Olson and Baroudi, 1982). 
Pearson's Methodology, which has received wide acceptance, 
uses a semantic differential technique to rate 39 factors 
which cause satisfaction or dissatisfaction with 
information systems. The 39 factors were identified based 
on extensive literature review and expert opinion. (Evans 
et al . , 1988). 

4 . Selecting Articles Dealing With User Satisfaction 

The compilation of 'articles for this study obtained 
from the search found 96 as possibly dealing with user 
satisfaction or system effectiveness. These then were the 
studies initially reviewed by this study and are listed in 
Appendix A. 

a. User Satisfaction Studies 

Reading the 96 articles in Appendix A yielded 40 
articles dealing with user satisfaction or one of its aliases 
(as described in the preceding section) . The selected studies 
employed user satisfaction or promoted its' employment as a 
measurement tool for some aspect of information system 
effectiveness. These included empirical studies on system 
effectiveness and methodological prescriptions for the 
employment of user satisfaction in measuring system 

effectiveness. These studies are listed in Appendix B. 
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b. Alternative Measures of Effectiveness 



Appendix C lists the 19 studies included in this 
section. These alternative measures of effectiveness are 
described in the preceding section. 

c. Miscellaneous Literature 

Appendix F contains the 29 remaining studies 
reviewed. Included here are: literature reviews, essays, 

prescriptive methodologies, frameworks and tools; and 10 
articles not specifically dealing with effectiveness 
measurement or were too unclear as to method or objective to 
evaluate properly. 

B. REVIEWING THE USER SATISFACTION STUDIES 

The 40 user satisfaction studies in Appendix B were 
evaluated with respect to the following criteria: 

1. The conventional wisdom assertion (CWA) . 

2. Was the Justification Supported? 

3. References rejecting alternative measures of 
effectiveness . 

1 . The Conventional Wisdom Assertion (CWA) 

The idea of documenting the CWA occurred after the 
initial review of all articles. The impetus came from the 
following observations. One, that there were articles which 
directly claimed that user satisfaction was the generally 
accepted or prevalent method for measuring IS effectiveness. 
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Two, that often the only evaluation measure mentioned or 
considered was user satisfaction, i.e., possible alternative 
measures of IS effectiveness did not receive mention let alone 
consideration . 

Determining which studies made the CWA category was 

then necessary. There were varying degrees of subtly 

encountered when trying to apply the CWA classification, from 

merely implied to outright claims of prevalence or general 

acceptance. The outright claims were relatively easy to 

identify, as most made the outright claim in the Abstract or 

Introduction. Some examples are: 

The measurement of how satisfied a user is with his or her 
information system (user information system or UIS) has 
become a pervasive measure of the success or effectiveness 
of an information system. (Baroudi and Orlikowski, 1988). 

To measure system success, researchers have mainly relied 
on either of two proxies: user satisfaction or system use. 
(Barki and Huff, 1985) 

User information satisfaction is the preferred indicator 
of system success . . . (Tait and Vessey, 1988) 

One of the most widely used methods of evaluating system 
effectiveness is the user satisfaction survey." "... our 
rather wholesale acceptance of these [user satisfaction] 
evaluation tools. (Hufnagel, 1990) 

The primary consideration in evaluating Management 
Information Systems (MIS) effectiveness is that it 'must 
be evaluated on the criteria of importance to the users' . 
(Lucas, 1978) 

user satisfaction with MIS . . . provides the most useful 
assessment of system effectiveness. (Hamilton and 
Chervany, 1981) 

Table I of Appendix E contains those studies that were 
identified as having made the CWA. Appendix D, Annotated 
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Description of User Satisfaction Articles, contains the 
selected documentation evidence used for assigning the CWA 
classification to a study. Included in the description are 
quotations and page references. For outright claims that user 
satisfaction is the prevalent or accepted method of measuring 
system effectiveness, this documentation serves as complete 
evidence for the CWA classification decision. 

Those that only implied the CWA were considerably more 
difficult to notice and especially difficult to document. This 
may seem contradictory to the notion of conventional wisdom. 
However, when taken to the extreme, conventional wisdom is an 
assumption by its user and, as such, is easily overlooked and 
difficult to pinpoint by an outside party searching for 
specific documentation. Additionally, the purpose of some 
studies was to further develop or test a particular user 
satisfaction tool. Generally, for these cases it was often 
necessary to examine the Introduction, Methodology (for 
description of the measurement tool) , and Conclusions sections 
to determine the extent of an author's assumption regarding 
his intended and actual employment of the tool as a measure of 
effectiveness, with respect to the CWA. 

The annotated description of User Satisfaction 
Articles, Appendix D, contains relevant quotations concerning 
the CWA. It should be noted that the quotations from some 
studies do not clearly indicate a conventional wisdom 
statement. However, they do at least suggest an attitude of 
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acceptance of the measure, which, when combined with other 

statements in the article, provided enough circumstantial 

evidence for it to be included in the CWA category. The 

circumstantial evidence which lead to the CWA judgement was 

mainly obtained from reading the Introduction, Methodology and 

Conclusions sections with respect to intent, actual employment 

and information provided about the system by the user 

satisfaction tool. The following are some examples: 

In information systems (IS) research, user satisfaction (or 
user information satisfaction) is considered to be a key 
construct for assessing system performance. (Barki, 1990) . 

While acknowledging the importance of economic analysis of 
MIS value, researchers respond to the shifting emphasis 
form efficiency to effectiveness by focusing either on MIS 
usage or user perceived effectiveness. (Srinivasan, 1985) 

2. Justification Supported 

Justification Supported was the classification given 
to studies which properly applied user satisfaction as an 
effectiveness measure. When determining whether the employment 
of user satisfaction was justified the criterion of this study 
was the degree to which the tool was to measure effectiveness. 
Was it as a "relative" measure or as an "absolute" measure of 
system effectiveness? In other words, a "better" or "more" 
effective system rather than that the system "was" or "was 
not" successful. The choice of relative or absolute measure 
was selected as best describing the myriad of approaches to 
employment user satisfaction has in the literature, from a 



28 



single item (factor) measure of early works, to multi-factor 
measures employed in a triangulation method. 

This study used as its basis for judgement, arguments 
from Denzin (1989) and Phillips (1971) . These were chosen for 
their information concerning the survey technique, its use and 
limitations . 

Denzin presents the theory of symbolic interactionism 

as the perspective from which to analyze research methods. 

Briefly summarized the concept of symbolic interactionism is: 

If human behavior is observable at two levels - the 
symbolic and the interactional - then central to 
understanding such behavior is the range and variety of 
symbols and symbolic meaning shared; communicated and 
manipulated by interacting selves in social situations. 
Society contributes two essential elements that reflect 
directly on concrete interactions: (1) the symbols; or 

various languages provided and communicated through the 
socialization process, and (2) the concrete behavioral 
settings in which the interaction occurs. (Denzin, 1989) 

With respect to this study, the symbolic level is 
interpreted as the opinion expressed by the user regarding his 
or her satisfaction with the system and the interactional 
level is that of the observed or measured behavior, 
effectiveness or performance, of the user himself. Regarding 
the survey technique, Denzin states that " Users of the survey 
seldom combine symbols with interaction ..." (Denzin, 1989). 

Phillips, when discussing interviews and 
questionnaires states that: 

. . . inferences about nonverbal behavior which are based on 
nonverbal responses are more subject to distortion and 
errors than are inferences about future or past nonverbal 
behavior which are based on the observation of such 
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behavior. That is, it is easier to 'lie with words' than 
to 'lie with acts' (Phillips, 1971). 

Here again the author questions the ability of the 
survey technique to draw inference to nonverbal behavior, 
i.e., action. Furthermore, only some of the user satisfaction 
authors. Table VIII, have addressed questions of measurement 
validity and reliability. However, one particular type 
proposed by Phillips that seems pertinent has not, convergent- 
descriminant validity. This is the ability to invalidate 
measures by showing that they "correlate poorly with some 
criterion but also by showing not only that they correlate 
highly with some conceptually simpler variable, such as the 
tendency to respond true or in a socially desirable manner at 
all times." (Phillips, 1971). 

The basic premise in the current study is drawn from 
arguments by Denzin and Phillips, that is, that users 
responses on a user satisfaction survey regarding information 
system effectiveness (opinion) are not the same as measuring 
performance as a result of an information system. This study 
makes a differentiation between the relative measure, 
employment as a descriptor or supplementary measure of the 
system and the absolute measure, success or failure of the 
system as its means of classifing opinion (relative) and 
action (absolute) . The delineation was made by drawing an 
anology to the opinion-performance arguements of Denzin and 
Phillips. Relative measures were viewed as anologous to 
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opinion, i.e. they can enhance or supplement objective or 
primary measures but cannot stand on their own. Absolute 
measures, on the other hand, have the necessary and sufficient 
conditions to stand on their own. The relative measures were 
designated Justification Supported while the absolute were 
not . 

Deciding if a particular study's employment of user 
satisfaction was justified, this study first looked at whether 
the article being reviewed employed a multiple-criteria 
method, i.e., user satisfaction was supplementing information 
from another measurement technique. This ancillary use was 
determined to be a justified use of the user satisfaction tool 
and classified as a relative measurement thus was a case of 
Justification Supported. There was considerable latitude in 
applying the multi-criteria test. This broad interpretation 
was because this author was not skilled in determining the 
validity of the various research methods encountered and 
wished to be as conservative as possible in assigning 
restrictions to the method, thereby providing the researchers 
the widest possible latitude in their employment of the user 
satisfaction tool. 

Studies which used user satisfaction as the sole means 
of measuring the system effectiveness constituted the other 
major category. It was necessary to determine the use of the 
user satisfaction tool's value, was it for relative 
qualification or judgement of the system or was it for an 
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absolute quantification or description of the system's success 
or failure. This was a particularly difficult assessment to 
make. As with the CWA, some authors stated directly how they 
intended to employ the tool, while others had conflicting 
statements in their articles which required analyzing the 
intent and conclusions drawn from the measure's employment. 
The following are e.xamples of the basic Justification 
Supported categories: 

Justification Supported; Yes, relative measure. 

concerned only with identifying difference in the 
affective response patterns of the two groups of 
performers (as opposed to arriving at a 'true' measure of 
value. (Hufnagel, 1990) 

Justification Supported: No, absolute measure. 

The measurement of how satisfied a user is with his or her 
information system (user information satisfaction or UIS) 
has become a pervasive measure of the success or 
effectiveness of an information system. (Baroudi and 
Orlikowski, 1988) 

Justification Supported: Yes, multiple criteria. 

The purpose of this study was to apply four methodologies 
to help understand and measure current information systems 
. . . The objective is to recommend an easy to use 
instrument for improving the effectiveness of information 
systems. (Evans et al . , 1988) 

Table I of Appendix D contains the determination data 

on Justification Supported. Additionally it contains the page 

number where relevant information leading to the judgement is 

located. Appendix D contains selected quotations which either 

provide direct evidence or substantial circumstantial evidence 
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documenting the study's documentation Justification Supported 
decision . 

Another category of data in this thesis was whether 
there was a specific reference to rejection of an alternative 
measure (s) of system effectiveness, see Table V, Appendix E. 
All that was required was for a study to include somewhere in 
its discussion and in its list of references an alternative 
measure. The point being, that there was an attempt made at 
selecting an appropriate method to measure system 
effectiveness . 
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IV. FINDINGS 



Appendix D contains an annotated description of each of 
the 40 user satisfaction studies evaluated here. Appendix E 
contains the detailed tables of findings for each aspect of 
this study. They are listed in chronological order by date of 
publication . 

Studies in Appendix B employed a form of user satisfaction 
measure to evaluate the effectiveness of an information 
system . 

In Appendix C, the studies used an alternative measurement 
tool to evaluate system effectiveness. 

A. CONVENTIONAL WISDOM ASSERTION 

The first question for which an answer was sought was what 
number of articles assume user satisfaction is currently the 
conventional wisdom or prevalent method employed by the IS 
community when measuring the effectiveness of an information 
system . 

Table I of Appendix E condenses out of Appendix B the 
primary references cited by each author for choosing user 
satisfaction as a surrogate measure of system effectiveness 
and for those with none listed the conventional wisdom 
assertion (CWA) was assigned. Table II shows that the CWA is 
the only reference for 45% of the MIS studies that employ user 
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satisfaction as a measurement tool. This implies that MIS 
researchers presuppose not only that user satisfaction is the 
preferred method of measuring MIS effectiveness but that it 
has also been proven and accepted as such by the MIS 
community. This is certainly not the case as shown by Appendix 
C, which lists 19 studies that use alternative methods of 
measuring an information system's effectiveness. Additionally, 
it contains several references in MIS literature rejecting 
user satisfaction as an absolute or single value measurement 
of a system's effectiveness (Ahituv, 1980; Marsden and 
Pingrey, 1988; Hufnagel, 1990) on theoretical or 
methodological grounds. 

Additional distillation of the data in Table I at years 
1977 (Pearson) and 1983 (Bailey and Pearson) provides further 
insight into the genesis and evolution of the CWA. In 1977 
Pearson' s dissertation developed a tool for measuring user 
satisfaction. In 1983 Bailey and Pearson further refined the 
tool in a published article (Bailey and Pearson, 1983) . Evans 
et al . (1988) state that "Instruments to measure user 
satisfaction are scarce." Most of those that existed prior to 
an empirically derived methodology developed by Pearson have 
been heavily criticized. Now, examining the percentage of the 
CWA in studies from 1977 forward and 1983 forward it appears 
that the emergence of the CWA has it's origins with Pearson's 
1977 dissertation and it's widespread practice with the 
publishing of the 1983 Bailey and Pearson study. There is also 
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an increase in the number of empirical studies evaluating IS 
(Alavi et al . , 1989). 

Studies with a CWA of user satisfaction have grasped this 
tool as the an easy and reliable way to measure system 
effectiveness. Often they have ignored the limitations stated 
by Bailey and Pearson concerning its use. Bailey and Pearson 
propose that it be used for "relative" evaluation, i.e. 
comparison of and between systems. Many authors making the CWA 
imply and draw conclusions of user satisfaction's absolute 
measurement of system effectiveness. 

B. JUSTIFICATION SUPPORTED 

Table III contains the studies of Appendix B which have a 
justification for employment of user satisfaction that is 
supported by the reference literature, i.e., that the user 
satisfaction tool was used properly according to a cited, 
developed methodology and subsequent conclusions limited to 
the theoretical restrictions of the tool. The judgement 
supported perspective shows that over 40% of the studies 
employing user satisfaction have incorrectly applied the tool. 
The most common error was to wrongly imply an absolute value 
of system effectiveness from the level of user satisfaction, 
i.e., a successful or unsuccessful system, rather than a 
relative measure, which is a comparison between alternatives 
or situations. 
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Only 19 of the 40 user satisfaction studies properiy 
applied the user satisfaction tool. Those that did limited its 
range of measure to some aspect of relative effectiveness 
rather than employing it as an absolute measure. 

Eight of the 19 are multiple criteria evaluations where 
user satisfaction was employed as one of several effectiveness 
measures. The multiple criteria approaches in general centered 
around "the relationship between satisfaction and bottom line 
indicators of user and organization performance" (Bailey and 
Pearson, 1983) . These bottom line indicators are what this 
study considers absolute measures of effectiveness 
(performance) . 

When user satisfaction was employed as a single item 
measure it was most commonly used as a "tool for improving 
system design" (Bailey and Pearson, 1983), which is 
supplementary or descriptive and thus classified as a relative 
measure by this study. 

Further examination of the studies that use the CWA shows 
that 66% of them have also incorrectly applied the tool. The 
implication here is that not only have researchers not 
evaluated other methods of measurement as evidenced by the 
CWA, but that they have also not evaluated the user 
satisfaction tool itself and understood its limitations and 
restrictions in particular those citing Bailey and Pearson. 
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C. REFERENCES TO REJECTION OF ALTERNATIVE MEASURES 

Table V applies a breakdown similar to both CWA and 
Justification Supported with respect to studies which actually 
reference an article which supports the rejection of 
alternative measures to user satisfaction. Common statements 
such as "it is too difficult" or "too costly" without 
reference, forms another category of conventional wisdom, 
i.e., that alternative methods cannot be used for measuring 
system effectiveness. MIS authors using user satisfaction have 
polarized their view of MIS effectiveness measurement to the 
point that it consists solely of the user satisfaction tool. 

D. REJECTION OF USER SATISFACTION 

Various articles in MIS literature have rejected, whole or 
in part, user satisfaction as a surrogate measure of system 
effectiveness. Hall and Lincoln (1976) "considered the 
possibility of measuring user satisfaction as a surrogate 
measure for information utility . . . However it was felt that 
an index of user satisfaction would not have provided senior 
BAC-CAD management with the means to compare the effectiveness 
of information systems with that of other company resources." 
This early rejection of the measure conceptualizes the 
ultimate purpose of any absolute evaluation measure, which is 
an overall assessment. The value of which is either for 
judging the success or failure of the system or as a common 
measure to compare one system to another. In either case the 
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it can stand 



value of the measure is complete in itself, i.e., 
alone and convey valuable information, for justifying company 
resources or comparing completely separate information 
systems . 

The other use to which an evaluation measure is put is to 
compare the relative value within a single item context, i.e., 
better/worse rather than successful/ failure . Here again, 
critics have found fault with user satisfaction on both 
theoretical and methodological grounds. Many reject it saying 
"single item" measures are inadequate (barker and Lessig, 



1980; Hamilton 


and Chervany, 


1981/ 


Barki 


and 


Huff, 1985) . 


Additionally, others (Matlin, 


197 9/ 


Markus 


and 


Robey, 1983/ 


and Jarvenpaa 


et al., 1985) 


argue 


that 


user 


satisfaction 



measures often lack theory and that the results are subject to 
influence by other factors within the system and therefore it 
is an inconsistent measure and a meaningless tool for 
evaluation . 
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V. CONCLUSIONS 



The evolution of user satisfaction as a surrogate measure 
for IS effectiveness suffers from two deficiences. The first 
is, that in 45 percent of the studies reviewed, it was the 
only method considered, which this study has labled the 
conventional wisdom assertion (CWA) . The second stems in part 
from the CWA, that in over 65 percent of the studies the tool 
was improperly applied. 

The MIS community's desire for credibility as a research 
field has led in part to the CWA. A credible user satisfaction 
measurement tool was lacking before that introduced by Pearson 
(1977) and developed by Bailey and Pearson (1983) . This prior 
void and the lack of empirical studies in MIS research (Alavi 
et al . , 1989) contributed in large part to a blind acceptance 
of the user satisfaction tool as a surrogate measure for 
system effectiveness by those making the conventional wisdom 
assertion. Many in the MIS community grasped the Bailey and 
Pearson tool without regard to its foundation, often 
improperly applying it. By failing to question the newly 
developed tool and even failing to understand its limitations, 
those making the CWA have done a disservice to the MIS 
community and MIS as a whole. Citing Keen (1980) Alavi et al . 
(1989) in their review of MIS research say "Even emerging 
fields such as MIS have a sense of history; those who cut 
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themselves off from curiosity and reflection are likely to 
atrophy . " 

Additionally, the MIS community's need for empirical 
research (Alavi et al., 1989) facilitated the acceptance of 
user satisfaction as an empirical tool. There were no 
instances of the CWA in MIS studies prior to Pearson's 1977 
dissertation and by Bailey and Pearson's 1983 study 57 percent 
of those that employed user satisfaction made the CWA. This 
highlights the blind acceptance of it by many researchers in 
the MIS community. 

The second problem is that of improper application. 
Basically the user satisfaction tool, as developed by Pearson 
and modified by others (Bailey and Pearson, 1983; Ives et al . , 
1983) is intended to measure satisfaction as an indicator of 
the relative effectiveness of an information system by means 
of "his/her reaction to the [user satisfaction] factor 
relative to the perceived information requirements" (Bailey 
and Pearson, 1983). If the user satisfaction value is higher 
for one case than another the higher valued system will be 
more effective than a lower valued one, whether it is the same 
system which was or will be modified or a totally sepatate 
one. It is not able to evaluate the absolute effectiveness of 
an information system. That would require a comparison between 
inputs and outputs of the system (s) . As Marsden and Pingrey 
(1988) point out "the key end user, the firm, will be 
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'satisfied' when its investment decision, that of the 
information system, contributes to the success of the firm." 

Therefore, if properly applied employing user satisfaction 
as a surrogate measure for system effectiveness may indicate 
"more" or "less" effective systems. However, 40 percent of all 
user satisfaction studies reviewed and 57 percent of those 
which were conducted in 1983 and later (Bailey and Pearson) 
improperly applied the tool. This improper application is due 
to its employment as an absolute measure of system 
effectiveness. Combined with the conventional wisdom 
assertion, the implication is that MIS research using user 
satisfaction is on uncertain ground. Not only is the 
conventional wisdom assertion made by many authors (tablel) 
not justified but when employed user satisfaction is 
improperly applied nearly half the time. The researchers have 
not only accepted the surrogate unquestioningly but they have 
also not bothered to understand and correctly apply it. This 
has a negative effect on the MIS community which is seeking 
credibility as a scientific research field (Alavi et al., 
1989) and for the justification of the financial investments 
being made in information systems (Fortune, 26 May 1990) . 

The intent of this study was not to determine the 
scientific validity of user satisfaction as a surrogate 
measure of effectiveness. It was however, intended to shed 
some light on its evolution and use in MIS literature. 
Examining studies measuring system effectiveness it was 



42 



determined not only that methods other than user satisfaction 
are "alive and well," thus dispelling the conventional wisdom 



assertion, 

"hobgoblin" 



but also that user satisfaction may be the 
of effectiveness measurement (Soergel, 1976) . 
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VI . RECOMMENDATIONS 



The role of MIS research is to increase our understanding 
of information systems, thereby allowing prediction of 
outcomes and consequences of the various factors influencing 
an information system. To do so researchers must develop 
theories with which to guide research and research 
methodologies (Alavi et al., 1989) which provide a systematic 
investigation of these factors. The focus of this study was on 
the measurement of effectiveness of information systems. In 
particular, the evolution and use of user satisfaction as a 
surrogate measure of effectiveness. At issue was the 
employment of user satisfaction, an opinion, as a measure of 
effectiveness, an action. 

The articles read for this study and the subsequent 
findings suggest several things regarding the measurement of 
effectiveness of information systems. To begin with, the MIS 
community must define a standard framework of information 
system effectiveness. The wide variety of effectiveness 
definitions encountered made it difficult to assess the 
validity of any given measure. Additionally, a "systematic 
investigation" is difficult in that findings are not 
replicated and verified, and methodologies are not advanced by 
refinement . 
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Next, theories should be developed and tested. The fact 
that almost half of the user satisfaction studies applied the 
CWA and that 40% improperly applied the tool indicates that 
the field is not thinking about the theory behind the 
relationship. The difficulty that the academic field of 
management science has found linking satisfaction with 
performance should draw into question an analogous 
relationship in MIS between user satisfaction and 
effectiveness . 

Additionally, social scientists have questioned the 
validity of the survey as a research method and in particular, 
as an accurate measure of behavior. Less than a quarter of the 
user satisfaction studies did any validity checks and only one 
questioned the validity of user satisfaction as a surrogate 
for effectiveness by testing for other possibilities. Instead 
the validity checks merely sought to find a correlation 
between user satisfaction and another attitudinal variable. 

The ubiquitous statements in the literature that 
intangibles cannot be accurately measured, are too difficult, 
too time consuming or too costly to obtain, often without 
reference suggest a general attitude exists in the MIS 
community regarding effectiveness measurement. Combined with 
the fact that the MIS community seems to have embraced the use 
of a subjective surrogate for the same ends, it makes one 
wonder about the motivation or competence behind the research. 
Are we further extending our knowledge of information systems 



45 



or, is it an easy way to publish and gain tenure? Was the 
research strategy selected the best choice or the easiest? 

The MIS community needs to provide valuable information 
concerning information systems to the user community so that 
they may invest their capital resources more effectively in 
information systems. As Marsden and Pingrey (1988) point out 
"the key end user, the firm, will be 'satisfied' when its 
investment decision, that of the information system, 
contributes to the success of the firm." 
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serious questions about the validity of employing user 



sat . 


ratings as measures of system effectiveness." 


CWA: 

p. 437 


"One of the most widely used methods of 
evaluating system effectiveness is the user 
satisfaction survey." 


JS; 

p. 441 


"Sure the hypothesis being tested (H3) is 
concerned only with identifying the differences 
in the affective response patterns of two groups 
of performance (as opposed to arriving at a 
"true" measure of value)." 


RR Alt: 


None 



Baroudi, 


J.J., and W.J. Orlikowslci, "A Short Form Measure of 



User Information Satisfaction: A Psychometric Evaluation and 
Notes on Use, " Journal of Management Information Systems, 
Vol. 4, No. 4, (Spring 1988), pp. 44-59. 

Develops short form measure of user satisfaction 
from B/P 



CWA: 

p. 44 


"The measurement of how satisfied a user is with 
his or her information system (user into 
satisfied or unsatisfied) has become a pervasive 
measure of the success or effectiveness of an 
information system." 


JS: 

p 44-45 


"has become a pervasive measure of the success or 
effectiveness of an information system" 


RR Alt: 


None 



60 



Doll, W.J. and G. Torkzadeh, "The Measurement of End User 
Computing Satisfaction, " Management Information Systems 
Quarterly, (June 1988), pp . 259-274. 



instrument for measuring end user satisfaction 



CWA: 


No 


JS: 

p. 259 


"End user computing satisfaction (EUCS) is a 
potentially measurable surrogate for utility in 
decision making." 


RR Alt: 

p. 259 


"decision analysis approach is generally not 
feasible (Gallagher, 1974, Nolan and Seward, 
1974) " 



Evans, et al . , "An Instrument for Measuring Effectiveness of 
Information Systems," Computers and Engineering, Vol. 14, 

No. 3, (1988), pp. 227-236. 



Multi 


Criteria approach to understand and measure 
information systems 
information flows 
user needs 

utility of information 
user satisfaction 



alone and in combination are useful for improving 
information systems 

CWA: No 

JS: "After identifying dissatisfied individuals on 

p. 234-235 the Pearson NSI matrix, studying results of the 





printouts for individuals from the Krobock 
Methodology can help to show where improvements 
can be made in specific sources of information to 
increase the satisfaction of these individuals." 


p. 228 


"alone and in combination are useful for 
improving information systems" 


RR Alt: 


NA 



61 



Gallupe, R.B., and G. DeSanctis, "Computer-Based Support for 
Group Problem Finding: An Experimental Investigation," 
Management Information System Quarterly, Vol. 12, No. 2, 
(June 1988), pp . 277-296. 

Examines Group DSS on group discussion quality and 
individual perceptions 

"confidence and satisfaction . . . lower in GDSS supported 



groups 


than in non-supported groups" 


CWA: 


NA 


JS: 

p. 284 


"dependent variables were the discussion 
performances and perceived reactions 


RR Alt: 


NA 



Gurmaraes, T., and Y.P. Gupta, "Measuring Top Management 
Satisfaction with the MIS Department," Omega International 
Journal of Management Science, Vol. 16, No. 1, (1988), pp. 

17-24 . 

"questionnaire to measure top management' s satisfaction 
with their MIS department" 



CWA; 

p. 18 


"researchers have traditionally used measures of 
user information satisfaction instead" 


JS; 

p . 18 


"How should one assess overall MIS department 
performance?" 


RR Alt: 

p. 18 


"Many of the variables involved in assessing an 
MIS department's importance are 
intangible ... Because of the difficulty of 
evaluating the economics of information systems." 



62 



Tait, P., and I. Vessey, "The Effect of User Involvement on 
System Success: A Contingency Approach," Management 
Information Systems Quarterly, (March 1988), pp. 91-108. 



"Path analysis was used to investigate both the direct 
effects of the contingent variables on system success 
and the effect of user involvement." 



CWA: 

pp. 95-96 


"assessing the satisfaction of the user with the 
information provided by the system is probably 
the most common approach to measuring system 
success . . . user information satisfaction is the 
preferred indicator of system success" 


JS: 

pp. 95-96 


"information satisfaction is the preferred 
indicator of system success and is the measure 
used in this study to define successful systems 
development " 


RR Alt: 


None 



Miller, J., and B.A. Doyle, "Measuring the Effectiveness of 
Computer-Based Information System in the Financial Sector, " 
Management Information Systems Quarterly, Vol. 11, No. 1, 
(March 1987), pp. 107-124. 

instrument for measuring information system's 
effectiveness 



CWA: 

p. 108 


"The instrument used in this research consists of 
38 items, 24 of which derive from Pearson, 12 
from Alloway . " 


JS: 

p. 107 


"the effectiveness of the overall information 
system's effort ... is evaluated using an 
instrument based on previous work" 


RR Alt: 


No 



63 



Aldag, R.J., and D.J. Power, "An Empirical Assessment of 
Computer Assisted Decision Analysis, " Decision Sciences , 
Vol. 17, No. 14, (Fall 1986), pp . 572-588. 

lab study with multiple criteria 



CWA: 


NA 


JS: 

p. 576 


"Individuals ... will exhibit more confidence in, 
and satisfaction with" 


RR Alt: 


NA 



Baroudi , 


J.J., M.H. Olson and B. Ives, "An Empirical Study 



of the Impact of User Involvement on System Usage and 
Information Satisfaction," Communications of the ACM, Vol. 
29, No. 3, (March 1986), pp . 232-238. 

"that user involvement leads to system usage and/or 
information satisfaction" 



CWA: 

p. 233 


"user satisfaction. Generally these constructs 
are assumed to be indirect indicators of improved 
decision making performance" 


JS: 


See CWA 


RR Alt: 


None 



Franz, C.R., and D. Robey, "Organizational Context, User 
Involvement, and the Usefulness of Information Systems, " 
Decision Sciences, Vol. 17, No. 3, (Summer 1986), pp. 329- 
355. 

user involvement is positively related to user' s 
perceptions of system usefulness 



CWA: 

p. 335 


cites Bailey and Pearson (1983) and Ives, Olson 
and Baroudi (1983) 


JS: 

p. 329 


"related positively to user's perceptions" 


RR Alt: 


None 



64 



Franz, C.R., D. Robey, and R.R. Koeblitz, "User Response to 
an Online Information System: A Field Experiment," 
Management Information Systems Quarterly, (March 1986), pp . 
29-42 . 

"impact of information systems on user attitudes and 
perceptions of job characteristics" 



CWA: 

p . 32 


"There seems to be a general consensus that 
favorable attitudes are associated with 
successful implementation." 


JS: 

p. 32 


Hi "... significant impact" 
H2 "... significant impact" 


RR Alt: 


None 



Martin, M.P. and J.E. Trumbley, "Measuring Performance of 
Automated Systems," Journal of Systems Management, (February 
1986) , , pp . 7-17 . 

"measuring success of automated systems . . . systematic 
structure for measuring" 



CWA: 


No 


JS: 

p. 14 


"Kucher's )cey concept here is that of comparison" 


RR Alt: 


NA MC 



65 



Barki, H., and S.L. Huff, "Change, Attitude to Change and 
Decision Support System Success, " Information and 
Management, Vol . 9, (1995), pp. 261-268. 

relationships between the extent of changes caused 
by the systems to users' work environments and the 
users' attitudes towards work related changes 

concludes DSS users are more satisfied with and use 
more extensively those DSS that bring change than 
those that do not 



CWA: 

p. 262 


"To measure system success, researchers have 
mainly relied on either of two proxies: user 
satisfaction or system use" 


JS: 

p. 262 


"The present study was conducted to develop 
empirical evidence concerning . . . DSS success" 

"User satisfaction is a more appropriate proxy 
[for measuring system success]" 


RR Alt: 

p. 262 


Yes 



Hogue, J.T., and H.J. Watson, "An Examination of Decision- 
Makers' Utilization of Decision Support System Output, " 
Information & Management, Vol. 8, No. 4, (August 1985), pp. 
205-212 . 

Examines DSS contribution to decision making process 
Dss users are highly satisfied with their systems 



CWA: 


No 


JS: 

p. 211 


"... the evaluation of user satisfaction, or the 
measurement of the value or success of the DSS" 


RR Alt: 


NA 



66 



Srinivasan, A., "Alternative Measures of System 
Effectiveness: Associations and Implications," Management 
Information Systems Quarterly, Vol . 9, No. 3, (September 
1985), pp. 243-253. 

Reexamines the relationship between user perceived 
effectiveness measures (user satisfaction) and 
behavioral measures (use) 

no positive association found, contrary to much of 
existing MIS research 



CWA: 

p. 243 


"While acknowledging the importance of economic 
analysis of MIS value, researchers responded to 
the shifting emphasis from efficiency to user 
effectiveness by focusing either on MIS usage or 
user perceived effectiveness [satisfaction]. Much 
of the MIS literature of late uses one of the 
other . " 


JS: 

p. 243 


"Measuring and Improving Information System 
Effectiveness/Productivity ... Approaches that 
have been advocated include . . . user 
satisfaction" 


RR Alt: 


Yes 



Baily, J.E., and S.W. Pearson, "Development of A Tool for 
Measuring and Analyzing Computer User Satisfaction, " 
Management Science, Vol. 29, No. 5, (May 1983), pp. 530-545. 

tool for measuring and analyzing computer user 
satisfaction 



CWA: 

p. 530 


"It is well recognized that productivity ... 
Further it is argued that utilization is directly 
connected too the user communities' sense of 
satisfaction ..." 


JS: 

p . 530 


"it has been argued that user satisfaction is 
correlated to information system utilization and 
system success" 


RR Alt: 


None 



67 



Ives, B., M.H. Olson, and J.J. Baroudi, "The Measurement of 
User Information Satisfaction," Comm of ACM, Vol. 26, No. 
10, (October 1983), pp . 785-793. 

Produces a short form measure of user satisfaction 



CWA: 

p. 785 


"Satisfaction of users with their information 
systems is a potentially measurable, and 
generally acceptable, surrogate for utility in 
decision making." 


JS: 

p. 785 


"MIS provides a meaningful 'surrogate' for the 
critical but unmeasurable result of an 
information system, namely, changes in 
organizational effectiveness." 



RR Alt: Yes 

p. 785 

King, W..R., and B.J. Epstein, "Assessing Information System 
Value: An Experimental Study," Decision Sciences, Vol. 14, 
No. 1, (January 1983), pp. 34-45. 

manager assessments of the importance of a Hubritis 
are used to assess information system value. 



CWA: 


No 


JS: 

p. 34 


"We want to determine whether practicing 
managers' overall evaluations of 'total systems' 
are consistent with those that would be derived 
from linear models that incorporate the managers' 
evaluations in terms of information attributes." 



RR Alt: NA 



68 



Mahmood, M.A., "Development and Validation of a Tool for 
Measuring and Analyzing Decision Support Systems User 
Satisfaction," pp. 171-173. 



user 


satisfaction is a measure for DSS 


CWA: 

p. 171 


"User satisfaction is a good measure of system 
effectiveness. Furthermore, it is argued that use 
of a system is directly related to the user 
communities' sense of satisfaction. Measuring and 
analyzing user satisfaction has been of great 
interest ever since the dawn of the computer 
era . " 


JS: 

p. 171 


"user satisfaction is a good measure of system 
effectiveness" 


RR Alt: 


None 



Epstein, B.J., and W.R. King, "An Experimental Study of the 
Value of Information," Omega, Vol. 10, No. 3, (1982), pp. 

249-258 . 

multidimensional attribute approach to assess the 
value of information 



CWA: 


No 


JS: 

p. 255 


"predictions of total system evaluations were 
made on the basis of self-explicated attribute 
levels and weightings. These predictions were 
then compared with the actual overall system 
assessments that were made by the respondents." 


RR Alt: 


None 



69 



Ginzberg, M.J., "Finding an Adequate Measure of OR/MS 
Effectiveness," Interfaces, Vol . 8, No. 4, (August 1978), 
pp. 59-62. 



user expectations as predictors of project success 
or failure 



CWA: 

p. 467 


"The method for measuring . . . success . . the 
literature has suggested both attitudinal (eg. 
user satisfaction) ..." 


JS: 

p. 463 


"There are basis in the literature on MIS 
implementation and in other fields to support the 
contention that expectations formed early in an 
MIS development project should be related to the 
eventual success or failure of that project." 


RR Alt: 


None 



Hamilton, S., and N.L. Chervany, "Evaluating Information 
System Effectiveness - Part I: Comparing Evaluation 
Approaches, " Management Information Systems Quarterly, 
(September 1981), pp. 55-69. 

compares evaluator (user, MIS development, internal 
audit, and top management personnel) viewpoints on 
system effectiveness 



CWA: 

p. 79 


"User satisfaction with the MIS . . . provides the 
most useful assessment of system effectiveness. 
Substantial evidence supports the validity ..." 


JS: 

p. 84 


"the use of a standardized validated instrument 
to measure system success" 


RR Alt: 


None 



70 



Larcker, D.F., and P.V. Lessig, "Perceived Usefulness of 
Information: A Psychometric Examination," Decision Sciences, 
Vol. 11, No. 1, (January 1980), pp . 121-134. 

Instrument for measuring perceived usefulness 



CWA: 

p. 121 


"... the ability of MIS designers to specify 
information that is perceived as useful by 
managers is instrumental to the success (or 
failure) of the system" 


JS: 

p. 121 


(see CWA above) 


RR Alt: 


None 



Deese, D.R., "Experiences Measuring User Satisfaction, 
Proceedings of the Computer Measurement Group of ACM, 
Dallas, (December 1979), pp. 59-66. 

Employs Pearson's user satisfaction tool to 
supplement information on FEDSIM review of data 
processing installations 



CWA: 


NA 


JS: 

p . 59 


"to supplement information ..." 


RR Alt: 


NA 



Neumann , 


S., and E. Segev, "A Case Study of User Evaluation 



of Information Characteristics for Systems Improvement, " 
Information & Management, Vol. 2, (1979), pp. 271-278. 

Questionnaire used to evaluate information system 



CWA: 

p. 35 


No 


JS: 

p.- 35 


"In addition to the relative contribution of 
information systems ..." 


RR Alt: 


No 



71 



Robey, D., "User Attitudes and Management Information System 
Use," Academy of Management Journal, Vol . 22, No. 3, 
(September 1979), pp . 527-538. 



attitudes are positively related to use 



CWA: 


No 


JS: 

p. 533 


"The primary objective of this research was to 
relate the objective measures of system use to 
attitudes . " 


p. 537 


"Strong positive relationships have been 
demonstrated between specific user attitudes and 
actual use of an MIS. It does not necessarily 
follow, however, that attitudes cause behavior." 


RR Alt: 


None 



King, W.R. and J.I. Rodriguez, "Evaluating Management 
Information Systems, " Management Information Systems 
Quarterly, Vol. 2, No. 3, (September 1978), pp. 43-51. 

multiple criteria framework for evaluating MIS 



CWA: 


No 


JS: 

p. 45 


"These form broad areas of value assessments - 
attitudes, value perceptions, information usage 
and decision performance - may be measured at 
various stages of the design and development 
process to evaluate the MIS." 


RR Alt: 


NA 



72 



Lucas, H.C., Jr., "Application and 

Implementation/Unsuccessful Implementation: The Case of a 
Computer-Based Order Entry System," Decision Sciences, Vol. 
9, No. 1, (January 1978), pp . 68-79. 

"Successful implementation would be affected by the quality 
of service and the personal costs and benefits to the users' 



CWA: 


No 


JS: 

p. 72 


"Satisfaction, the indicator of successful 
implementation. " 


RR Alt: 

p . 69 


Yes 



Robey, D., and R.L. Zeller, "Factors Affecting the Success 
and Failure of an Information System for Product Quality, " 
Interfaces, Vol. 8, No. 3, (February 1978), pp. 70-75. 

analyze the adoption and rejection of the same 
information system by two similar departments of the 



same 


company 


CWA: 


No 


JS: 

p. 71 


"Responses show the user' s perception of how 
favorable the system is for each area of concern. 
Questionnaire data can thus provide evidence of 
specific user problems." 


p. 73 


"while we cannot say that attitudes were the 
cause of behavior..." 



73 



Dickson, G.W., J.A. Senn and N.L. Chervany, "Research in 
Management Information Systems: The Minnesota Experiments," 
Management Science, Vol . 23, No. 9, (May 1977), pp . 913-923 

"The Minnesota Experiments . . . examine the significance of 



various 


information system characteristics on decision 
activity . " 


CWA: 


NA 


JS: 

p. 917 


"In addition to discussion quality, measures of 
user evaluation of the information systems used 
were taken ..." 


RR Alt: 


NA 



Edstrom, A., "User Influence and the Success of MIS Project: 
A Contingency Approach," Human Relations, Vol. 30, No. 7, 
(1977), pp. 589-607. 

"investigate alternative specifications of the relationship 
between user influence and the success of the management 

information systems" 

Outcome variables: Perceived success (i.e., user 
satisfaction) 



CWA: 

system 
p. 590 


"The perceived success of an information 

is, however, not the only possible criterion of 
success and it seems to us to be useful to 
investigate alternative aspects of success." 


JS: 

p. 589 


"the measures of outcomes [success] are 
perceptual measures ..." 


RR Alt: 


None 



74 



Schewe, C.D., "The Management Information user: An 
Exploratory Behavioral Analysis, " Academy of Management 
Journal, Vol . 19, No. 4, (December 1976), pp . 577-590. 

"Model exploring the relationships between management 
information system users' perceptions of their computer 
system, perceived variables exogenous to the system, 
attitudes and system usage." 



CWA: No 



JS: 

p. 579 



RR Alt: 



"According to the model, the beliefs about 
characteristics of the MIS are evaluated and 
result in some feeling of favorableness or 
unfavorableness about the information system." 

None 



Gallagher, C.A., "Perceptions of the Value of a Management 
Information System,” Academy of Management Journal, Vol. 17, 
No. 1, (March 1974), pp . 45-55. 



A method of determining the monetary value of an 
existing MIS report is based on user perceptions and 
the semantic differential technique 



CWA: No 



JS: 

p. 55 2 



"it seems fair to conclude that the EAB system 
was well thought of ... as a whole" 



RR Alt : Yes 

pp . 4 6, 47 



75 



Lucas, H.C., Jr., "Performance and The Use of an Information 
System," Management Science, Vol. 21, No. 8, (April 1975a), 
pp. 908-919. 

A study of user reactions, the use of information 
systems and an independent rating of systems 
quality . 



CWA: 


No 


JS: 

p. 207 


HI: "Favorable user attitudes are associated with 
favorable user perceptions of systems quality and 
independent ratings of systems quality." 


RR Alt: 


NA 



Swanson, 


E.B., "Management Information Systems: Appreciation 



and Involvement," Management Science, Vol. 21, No. 2, 
(October 1974), pp . 178-188. 

measurement of involvement and appreciation 



CWA: 


No 


JS: 

p. 1180 


"An increase in MIS appreciation of an individual 
will increase his inquiry involvement." 


RR Alt: 


No 



Powers, R.F., and G.W. Diclcson, "MIS Project Management: 
Myths, Opinions and Reality, " California Management Review, 
Vol. 15, No. 3, (1973), pp. 147-156. 

Research to identify "the correlates of MIS project 
success" 



CWA: 


No 


JS; 

p. 150 


"user satisfaction - attitudes of the managers 
receiving project products relative to how well 
their information needs were being satisfied." 


RR Alt: 


NA 
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APPENDIX E. TABLES OF FINDINGS 



TABLE I . STUDIES MAKING THE CWA 



Author 


Year 


Barki 


90 


Conrath/Mignen 


90 


Hufnagel 




Barooudi/Orlinkowski 


1 88 

1 1 


Guimaraes/ Gupta 


88 


Tati/Vessey 


1 '1 

, 88 

1 


Miller/Doyle 


87 ' 

1 


Baroudi /Olson/ Ives 


8 6 !■ 


Franz/Robey 


^ si ! 


Franz /Robey /Koebl it z 


86 1 


Barki/Huff 


85 


Srivinason 


85 


1 Bailey/Pearson 

1 


83 


Ives /01s on /Baroudi 


83 


II Mahmood 


83 


1 

Ginzberg 


81 


1 Hamilton/Chervany 


81 


barker /Vessig 
ii 


80 



TABLE II. BREAKDOWN OF THE CWA 



! All Yrs 


18/40 


- “ 1 

45% 


1 ' 83 + 


15/23 


65% 


' 77 + 

ii 


18/35 


51% 
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TABLE III. STUDIES WITH JUSTIFICATION SUPPORTED 



AUTHOR ■ YEAR 


r' ■ 

CWA 


Con rath /Mignen 


1 90 


Y 


Hufnagel 


90 


' ^ 1 


Evans, et al . 


: 88 


1 

; NA (MC) !, 


Gallupe/DeSanctis 


88 


NA (MC) 


Aldag/Power 


86 


^ NA (MC) 


Bar oudi/ 01 son /Ives 


86 


Y 


Franz/Robey 

1 


86 


Y : 


; Franz /Robey/Koeblitz 


86 


Y 


Mart in/Trumbley 


8 6 


NA (MC) 


Srinivason 


85 


Y 


Bailey/Pearson 


83 


Y 


I Ives/Olson/Baroudi 


83 


Y 


King/Epstein 


83 


N 


Deese 


79 


NA (FEDSIM) 


Nueman/Segev 


79 


N 


Robey 


79 


N 


King /Rodriguez 


778 


N 


Robey/Zellar 


78 


N 


Dickson/Senn/Chervany 


77 


NA (MC) 


Schewe 


76 


N 


Gallagher 


74 


N 


Lucas 


74 


N 


Swanson 


74 


N 


: Powers/Dickson 




73 


N 



TABLE IV. BREAKDOWN OF JUSTIFICATION SUPPORTED 



1 

All Yrs. 


24/40 


60% 


' 83 + 


13/23 


57% 


+CWA 


8/18 


44% 








' 83 + 

1 


8/13 


62% 








' 77 + 

1 ' — 


8/18 


44% 









78 



TABLE V. STUDIES CITING A REFERENCE REJECTING ALTERNATIVES 

(RR Alt) 



1 - 1 

AUTHOR 


YEAR 


CWA 


1 1 

Doll /Torkzadeh 


88 


N 


Barki/Huf f 


1 

1 

1 i 

LO 

OO 

1 


Y 


Srinivason 


85 


Y 


Ives/Olson/Baroudi 


83 


Y 


t Lucas ^ 


78 


N 


1 Gallagher 


74 


N 



TABLE VI. BREAKDOWN OF RR ALT 



All Yrs 


1 - ^ 

6/32 ; 19% 


+CWA 


3/18 


17% 


' 83 + 


3/13 


23% 


' 77 + 


3/18 


1 . 

17% 1 ; 








(T 

I 

!i 
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TABLE VII. NAMES, DEFINITION AND CONSTRUCT OF USER 
SATISFACTION VARIABLES 



STUDY ' VARIABLE; DEFINITION 

YEAR CONSTANT 



!' 1 . 


Powers/ Dickson 


user satisfaction: attitudes of managers 




' 73 


not descrioed 




Swanson 


appreciation: users beliefs about relative value 


l| 


' 74 


not described 


3 . 


Lucas 


1 

user attitudes: not defined 




' 74 


some scale not described ; 




Gallagher 


user perceptions: not defined 




' 74 


semantic differential j 


5. 


Schewe 


! 

user perceptions, attitudes: not defined 




' 76 


not described 


II 

6. 


Edst rom 


perceived success: not defined 




' 77 


com^plete success - complete failure 


7. 


Dukson/Senn/Chervany 


user evaluation: not defined 


1 


' 77 


not described 


8. 


Robey/Zel ler 


attitudes: not defined 




' 78 


Shul tz-Slevin 


9. 


Lucas 


user perceptions: modified JDI 


) 


' 78 


Shul tz-Slevin 


1 10. 


King /Rodriguez 


value perceptions: "direct assessment" of MIS 


1 


' 78 


Shultz-Slevin 


ij 

! 11 . 


Robey 


user attitudes: not defined 


II 


' 79 


Shultz-Slevin 


' 12. 


Neumann/Segev 


user evaluation: accuracy, content, frequency, recency 




' 79 


questionnaire 


1 13. 


Deese 


user satisfaction: * 




' 79 


Pearson method 


14. 


Larkin/Lessig 


perceived useableness/perceived importance | 




' 80 


questionnaire - 7 pt scale 


15. 


Hamilton/Chervany 


user satisfaction; * 




' 81 


Pearson method 


^ 16. 


Ginzberg 


user satisfaction 




' 81 


Shultz-Slevin 


17. 


Epstein/King 


satisfaction/value perception: 10 attributes 




' 83 


guest ionnarie, 5 pt scale 


18. 


Mahmood 


user satisfaction: * 




' 83 


Pearson m.ethod 


19. 


King/Epstein 


satisfaction/value perception; 10 attributes 




' 83 


questionnaire, 5 pt scale 


' 20. 


Ives/01 son/Baroudi 


user satisfaction: * 




' 83 


Pearson, Bailey/Pearson 


! 21. 


Bailey/Pearson 


user satisfaction: semantic differential factors 39 


1 


' 83 


Pearson method 


1 

; 


Srinivasen 


user satisfaction: * 




' 85 


Bailey /Pearson 


1 23. 


Hogue/Watson 


perceived value: satisfaction/diissatisfaction 


il 


'85 


questionnaire 7 pt scale 



80 



STUDY 

YEAR 


VARIABLE: DEFINITION 
CONSTANT 


: 24 , 3ar<./H^ff 

'| 


user satisfaction: ♦ 
Pearson Method 


.1 25. Mart i r. /Tr jrr.bo.y 


user accorrur.oaat ion : “ruxe of sat^sfiiing" 
^ Questionnaire - Libert scales 


,| 

'! 26. Frar.z/Rcbey /Koebx , ‘ 2 
1 '86 


• user att i tudes/percept ^ons 
Schul t z-Slevin 


1' 27. Franz/Robey 
1 '86 


perceivea usefulness: user satisfaction 
. Bailey and Pearson 


28. Baroudi /Olson/Ives 

t ' 86 


[ user satisfaction: * 
Bailey and Pearson 


29. Aldag/Power 
'86 


1 user perceptions: multiple item 
Likert scale 


30. Milier/Doyle 
'87 


user satisfaction: * 
Bailey and Pearson 


31. Tart/Vessey 
'88 


user satisfaction: 
Bailey and Pearson 


. 32. Gurmaraes/Gupta 

I ' 88 


user satisfaction: * 
Bailey and Pearson 


33. Gal lupe/DeSanct i s 
1 ' 88 


perceived reactions, satisfaction 
questionnaire, Likert scale 


34. Evans, et. al 
'88 


user satisfaction: * 
Pearson Method 


35. Doll/Torkzadeh 
! ' 88 


! 

user satisfaction: * 1 

Bailey and Pearson 


1 36. Baroudi/Orlikowski 

' ' 88 


user satisfaction: short form 13 factors * 
Bailey and Pearson 


37. Hufnagel 
' 90 


! 

user satisfaction: * 

Bailey and Pearson 


38. Conrath/Mignen 
' 90 


user satisfaction: * | 

Bailey and Pearson j 


39. Burkhard 
' 90 


! 

user satisfaction: * ' 

Bailey and Pearson short form 


1 40. Barki 

1 


!| 

user satisfaction: * i 

Bailey and Pearson 



see "USER SATISFACTION" in Section III, METHODOLOGY 
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TABLE VIII. STUDIES WITH VALIDITY/RELIABILITY CHECKS 



Hufnagel 


' 90 


Baroudi/Orlikowski 


1 

' 88 


Doll/Torkzadeh 


' 88 


Gurmaraes /Gupta 


' 88 


Tart /Vessey 


' 88 


Miller/Doyle 


' 87 


Bailey /Pear son 


' 83 


1 

Ives/Olson/Baroudi 


' 83 


Mahmood 


' 83 


t 

Larcker/Lessey 


' 80 
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